Classification of feature information into groups based upon similarity, and apparatus, image processing method, and computer-readable storage medium thereof

ABSTRACT

An apparatus extracts feature information from an object of image data. The apparatus registers the extracted feature information in a dictionary. The apparatus refers to the dictionary and determines a similarity between feature information registered in the dictionary and the extracted feature information. The apparatus does not use, of feature information to be registered in the dictionary, feature information not satisfying a predetermined evaluation criterion in similarity determination.

This application is a continuation of U.S. patent application Ser. No.13/932,686, filed Jul. 1, 2013 (allowed), the contents of which areincorporated herein by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image processing technique ofdetecting and managing an object in image data.

Description of the Related Art

As digital still cameras (to be also referred to as “DSCs” hereinafter)become popular, image data as many as several thousands or several tenthousands need to be handled nowadays. As one especially importanttechnique for the handling method, personal recognition is implementedby handling images based on a person's face. For example, a face regionincluded in an input image is detected in advance, information (to bereferred to as a “feature amount” hereinafter) obtained by analyzing thedetected face image is extracted, and the extracted feature amount isregistered. Note that the feature amount to be registered increases inaccordance with the number of images to be analyzed and the number offaces included in an image. A database in which a plurality of featureamounts are registered will be called a “dictionary” or “facedictionary”. Personal recognition becomes possible by collating anobtained face dictionary with a feature amount obtained by analyzing anewly input image.

Japanese Patent No. 3469031 discloses an arrangement for evaluating anddetermining whether a face newly detected from an image should beregistered in a face dictionary.

However, in Japanese Patent No. 3469031, incorrect determination mayoccur regardless of how to select a face to be registered in the facedictionary. The incorrect determination means a situation in which aperson, who has already been recognized, is generated as a new differentperson as a result of recognition.

An example of the incorrect determination will be described in detail.When many images are scanned, many image feature amounts are registeredin a dictionary. At this time, exceptional feature amounts are sometimesregistered in the dictionary even for a family as a result ofexceptional capturing. In such a case, a plurality of persons aregenerated even for the same person as identification results.

Also, in Japanese Patent No. 3469031, an object (for example, a passerirrelevant to the user) other than a target object may be registered inthe face dictionary. A situation in which an object other than a targetone is identified will be called “erroneous determination”.

Repetition of incorrect determination and erroneous determinationimpairs the personal recognition accuracy based on the face dictionary.

SUMMARY OF THE INVENTION

The present invention provides an image processing technique ofincreasing the determination accuracy based on a dictionary used forobject similarity determination.

To achieve the above object, an image processing apparatus according tothe present invention comprises the following arrangement.

That is, an apparatus comprises: an extraction unit configured toextract feature information from an object of image data; a registrationunit configured to register, in a dictionary, the feature informationextracted by the extraction unit; and a similarity determination unitconfigured to refer to the dictionary and determine a similarity betweenfeature information registered in the dictionary and the featureinformation extracted by the extraction unit, wherein, of featureinformation to be registered in the dictionary, feature information notsatisfying a predetermined evaluation criterion is not used indetermination by the similarity determination unit.

According to the present invention, the determination accuracy based ona dictionary used for object similarity determination can be increased.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the hardware arrangement of an imageprocessing apparatus;

FIG. 2 is a block diagram showing software for controlling the imageprocessing apparatus;

FIG. 3 is a flowchart showing personal recognition processing;

FIG. 4 is a flowchart showing details of face dictionary generation;

FIG. 5 is a view showing a display example of an image group in athumbnail format;

FIG. 6 is a view showing a display example of an image group in acalendar format;

FIG. 7 is a view showing the structure of a face feature amount;

FIG. 8 is a view showing a display example of an image group in athumbnail format;

FIG. 9 is a view showing the structure of a face dictionary;

FIG. 10 is a flowchart showing face dictionary customization proceduresaccording to the first embodiment;

FIG. 11 is a flowchart showing face dictionary customization proceduresaccording to the second embodiment;

FIG. 12 is a view showing an example of a smoothing filter;

FIG. 13 is a flowchart showing personal recognition processing includingface dictionary customization according to the third embodiment;

FIG. 14 is a flowchart showing execution of face dictionarycustomization according to the third embodiment;

FIG. 15 is a flowchart showing face dictionary customization proceduresaccording to the fourth embodiment;

FIG. 16 is a view showing an example of a UI used to manually input thefavorite rate;

FIG. 17 is a flowchart showing details of face dictionary generation;

FIG. 18 is a view showing an example of the structure of a category;

FIG. 19 is a flowchart showing face dictionary customization procedures;and

FIG. 20 is a flowchart showing face dictionary customization procedures.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will now be described in detailwith reference to the accompanying drawings.

First Embodiment

An embodiment of the present invention will be described toautomatically generate a layout output by using an input image group.This merely exemplifies a form of implementation, and the presentinvention is not limited to the following practice.

The embodiment will exemplify the arrangement of a personal recognitionmodule including a face dictionary, an arrangement for obtaining apersonal recognition result, and a hardware arrangement in which thepersonal recognition module runs. Embodiments of the present inventionare not limited to the following embodiments, and the personalrecognition module in the embodiments can also be implemented in a DSCor printer having a similar hardware arrangement.

<Description of Hardware Arrangement>

FIG. 1 is a block diagram showing an example of the hardware arrangementof an image processing apparatus according to the embodiment.

In FIG. 1, an image processing apparatus 115 is, for example, acomputer. A CPU (Central Processing Unit) 100 executes informationprocessing to be described in the embodiment in accordance withprograms. The CPU 100 loads a program stored in a secondary storagedevice 103 and the like to a RAM 102 and runs the program on the RAM102, thereby controlling of the entire image processing apparatusaccording to the present embodiment. A ROM 101 stores programs to beexecuted by the CPU 100. A RAM 102 provides a memory to temporarilystore various kinds of information when the CPU 100 executes theprograms. A secondary storage apparatus 103 such as a hard disk is astorage medium to save, for example, a database that saves image filesand image analysis results. Not the ROM 101 but the secondary storageapparatus 103 may store programs for executing information processing tobe described in the embodiment.

A display apparatus 104 is an apparatus which presents, to the user,various kinds of UIs (User Interfaces) to be described below, includinga processing result in the embodiment. The display apparatus 104 is, forexample, a display. The display apparatus 104 may have a touch panelfunction. A control bus/data bus 110 connects the above-describedbuilding elements to the CPU 100. The image processing apparatus 115also includes an input apparatus 105 such as a mouse or keyboard used bya user to input an image correction processing instruction and the like.

The image processing apparatus 115 may include an internal imagecapturing device 106. An image captured by the internal image capturingdevice 106 undergoes predetermined image processing and is stored in thesecondary storage apparatus 103. The image processing apparatus 115 mayload image data from an external image capturing device 111 connectedvia an interface (an IF 108). The image processing apparatus 115 alsoincludes a LAN (Local Area Network) IF 109. The LAN IF 109 may be wiredor wireless. The LAN IF 109 is connected to Internet 113. The imageprocessing apparatus 115 can also acquire an image from an externalserver 114 connected to the Internet 113.

A printer 112 for outputting an image and the like is connected to theimage processing apparatus 115 via an IF 107. The printer 112 is furtherconnected to the Internet 113 and can exchange print data via the LAN IF109.

FIG. 2 is a block diagram showing a software arrangement including theabove-described application according to the embodiment.

Image data acquired by the image processing apparatus 115 is normallycompressed in a compression format such as JPEG (Joint PhotographyExpert Group). Hence, an image codec unit 200 decompresses image databased on the compression format and converts it into image data (bitmapdata) in a so-called RGB dot-sequential bitmap data format. Theconverted bitmap data is transferred to a display and UI control unit201 and displayed on the display apparatus 104 such as a display.

When image data acquired by the image processing apparatus 115 is imagedata requiring no decoding processing, the processing by the image codecunit 200 is unnecessary.

The bitmap data is further input to an image sensing unit 203(application), and undergoes various analysis processes (details will bedescribed later) by the image sensing unit 203. Various kinds ofattribute information of the image obtained by the analysis processingare stored in the secondary storage apparatus 103 by a database unit 202(application) in accordance with a predetermined format. Note that imageanalysis processing and sensing processing will be handled in the samesense.

A scenario generation unit 204 (application) generates the conditions ofa layout to be automatically generated in accordance with variousconditions input by the user. A layout generation unit 205 performsprocessing of automatically generating a layout for arranging image datain accordance with the generated scenario.

A rendering unit 206 renders the generated layout into bitmap data fordisplay. The bitmap data serving as the rendering result is transmittedto the display and UI control unit 201, and its contents are displayedon the display apparatus 104. The rendering result is also transmittedto a print data generation unit 207, and the print data generation unit207 converts it into printer command data and transmits the command tothe printer 112.

A personal recognition processing sequence will be explained withreference to FIG. 3.

In step S300, the image processing apparatus 115 generates a facedictionary for performing personal recognition by collating face images.The face dictionary generation method will be described later. In stepS301, the image processing apparatus 115 recognizes the person of a faceimage by using the face dictionary generated in step S300 (face imagecollation). The recognizing method will be described later. In FIG. 3,the face dictionary generation step and face image collation step aredescribed separately for convenience, but the execution of them is notlimited to this. That is, collation may be performed while generating aface dictionary. A case in which face dictionary generation and faceimage collation are performed simultaneously will be explained.

Face dictionary generation step S300 will be described in detail withreference to FIG. 4.

In step S400, the image sensing unit 203 acquires an image data group.For example, the user connects, to the image processing apparatus 115,an image capturing apparatus or memory card which stores capturedimages, and loads the captured images from it, thereby acquiring animage data group. Images which have been captured by the internal imagecapturing device 106 and stored in the secondary storage apparatus 103may be acquired. Image data may be acquired via the LAN 109 from anexternal apparatus other than the image processing apparatus 115, suchas the external server 114 connected to the Internet 113.

After the image data group is acquired, it may be displayed on a UI, asshown in FIGS. 5 and 6. A display on the display apparatus 104 uponacquiring an image data group will be explained with reference to FIGS.5 and 6. For example, in FIG. 5, folders in the secondary storageapparatus 103 are displayed in a tree-shaped structure in a folderwindow 501. If the user selects a folder 502, thumbnails 503 of imagesare displayed in a preview window 504. In FIG. 6, a calendar isdisplayed in a preview window 601. If the user clicks a date portion602, images captured at the clicked date can be displayed in the previewwindow 601 in the same fashion as the thumbnails 503.

In step S401, the image sensing unit 203 acquires various characteristicinformation such as characteristic amounts including a face position inan image. Table 1 exemplifies characteristic amounts to be acquired andtheir information types.

TABLE 1 Sensing Classification Sensing Sub-classification Data TypeBasic Image Average luminance int Characteristic Amount Averagesaturation int Average hue int Face Detection Number of person's facesint Coordinate position int * 8 Average Y in face region int Average Cbin face region int Average Cr in face region int Scene Analysis Sceneresult char

First, the average luminance and average saturation of an entire image,which are basic characteristic amounts of an image (basic imagecharacteristic amounts), are obtained by a known method. For example,the R, G, and B components of each pixel of an image are converted intoknown luminance and color difference components (for example, Y, Cb, andCr components), and the average value of the Y components is calculated.For the average saturation, the Cb and Cr components are calculated foreach pixel, and the average value S is calculated by solving equation(1):S=√{square root over (Cb ² +Cr ²)}  (1)

The average hue AveH in an image may be calculated as a characteristicamount to evaluate the tone of the image. The hues of respective pixelscan be calculated using a known HIS transformation, and smoothed in theentire image, thereby calculating AveH. This characteristic amount neednot be calculated for an entire image. For example, an image may bedivided into regions having a predetermined size, and the characteristicamount may be calculated for each region.

Processing of detecting the face of a person serving as an object willbe described next. Various methods are usable as the person's facedetection method used in the embodiment. In a method described inJapanese Patent Laid-Open No. 2002-183731, first, an eye region isdetected from an input image, and a region around the eye region is setas a face candidate region. The luminance gradient and the weight of theluminance gradient are calculated for the face candidate region. Thesevalues are compared with the gradient and gradient weight of a presetideal reference face image. At this time, when the average angle betweenthe gradients is equal to or smaller than a predetermined threshold, theinput image is determined to have a face region.

In a method described in Japanese Patent Laid-Open No. 2003-30667, aflesh color region is detected from an image. A human iris color pixelis then detected in the flesh color region, thereby detecting theposition of an eye.

In a method described in Japanese Patent Laid-Open No. 8-63597, thematching level between an image and each of a plurality of face shapetemplates is calculated. A template having a highest matching level isselected. If the highest matching level is equal to or more than apredetermined threshold, a region on the selected template is set as aface candidate region. By using this template, the position of an eyecan be detected.

In a method described in Japanese Patent Laid-Open No. 2000-105829, anoise image pattern is set as a template. Then, an entire image or adesignated region of an image is scanned. A position that matches thetemplate most is output as the position of the nose. A region above thenose position in the image is assumed to be a region where the eyesexist. The eye existence region is scanned using an eye image pattern asa template, thereby obtaining a matching level. A set of pixels whosematching levels are higher than a given threshold is acquired as an eyeexistence candidate position. A continuous region included in the eyeexistence candidate position set is divided into clusters. The distancebetween each cluster and the nose position is calculated. A clusterhaving a shortest distance is decided as a cluster including an eye,thereby detecting the organ position.

As other methods of detecting a face and organ positions, known methodsare usable, including methods described in Japanese Patent Laid-OpenNos. 8-77334, 2001-216515, 5-197793, 11-53525, 2000-132688, 2000-235648,and 11-250267, and Japanese Patent No. 2541688.

As a result of the processing, the number of person's faces in eachinput image data and the coordinate positions of each face can beacquired. Once the coordinate positions of the face (face region) inimage data are known, the average Y, Cb, and Cr values of pixel valuesincluded in the face region can be calculated for each face region. Inthis manner, the average luminance and average color differences of theface region can be obtained. Further, scene analysis processing can beperformed using the characteristic amount of an image. The sceneanalysis processing can use techniques disclosed in, for example,Japanese Patent Laid-Open Nos. 2010-251999 and 2010-273144 filed by thepresent applicant. As a result of the scene analysis processing,identifiers for discriminating capturing scenes such as Landscape,Nightscape, Portrait, Underexposure, and Others can be acquired.

Note that the sensing information is not limited to that acquired by theabove-described sensing processing, and any other sensing informationmay be used. The sensing information acquired in the above-describedmanner is stored in the database unit 202. The save format in thedatabase unit 202 is not particularly limited. For example, a structureas represented by Table 2 is described using a general-purpose format(for example, XML: eXtensible Markup Language) and stored.

TABLE 2 First Layer Second Layer Third Layer Contents BaseInfo ImageIDImage identifier ImagePath File storage location ImageSizeWidthA Imagesize and width CaptureDateTime Capturing date & time SenseInfo AveYAverage luminance of image AveS Average saturation of image AveH Averagehue of image SenseType Landscape Scene type Portrait . . . Person FaceIDIdentifier of detected face Position Position in image AveY Averageluminance of face AveCb Average Cb of face AveCr Average Cr of faceUserInfo FavoriteRate Favorite rate of image ViewingTimes Number oftimes of viewing PrintingTimes Number of times of printing Event TravelTheme of image capturing Anniversary . . .

Table 2 represents an example in which pieces of attribute informationof each image are classified into three categories and described. Thefirst BaseInfo tag is information added to an acquired image file inadvance and representing the image size and capturing time information.This tag includes the identifier <ImageID> of each image, the storagelocation <ImagePath> where the image file is stored, the image size<ImageSize>, and the capturing date & time <CaptureDateTime>.

The second SensInfo tag is used to store the result of theabove-described image analysis processing. The average luminance <AveY>,average saturation <AveS>, and average hue <AveH> of an entire image andthe scene analysis result <SceneType> are stored. Information of a facedetected in the image is described in a <Person> tag. Information aboutthe face position <Position> and face colors <AveY>, <AveCb>, and<AveCr> of the person in the image can be described. A<FaceID> tag is anidentifier (identifier for identifying the same object) which associatesfeature amounts extracted from the face image with an image file. Thefeature amounts extracted from the face image will be described later.

The third <UserInfo> tag can store information input by the user foreach image. For example, <FavoriteInfo> describes the scale of“favorability” to be described later. <ViewingTimes> is the number oftimes of viewing by which the user opened and viewed the image.<PrintingTimes> is the number of times of printing by which the userprinted. <Event> is information which can be added as an attribute ofthe image. These are merely examples, and the user can describe anotherkind of information. Note that the image attribute information databasestorage method is not limited to the above-described one. Also, theformat is arbitrary.

In step S402, the image sensing unit 203 generates a normalized faceimage. The normalized face images are face images obtained by extractingface images existing in images with various sizes, orientations, andresolutions, and converting and cutting out them into faces having apredetermined size and orientation. The positions of organs such as aneye and mouth are important to perform personal recognition. Thus, thenormalized face image can have a size enough to reliably recognize theorgans. By preparing the normalized face images, feature amountextraction processing in next step S403 need not cope with faces ofvarious resolutions.

In step S403, the image sensing unit 203 extracts face feature amountsfrom the normalized face image generated in step S402 (face featureamount extraction unit). The contents of the face feature amounts willbe described in detail. The face feature amounts are vector quantitieseach obtained by extracting at least one piece of information unique tothe face, such as an organ or outline. The face feature amounts are, forexample, pieces of individual information such as the sizes ofrespective organs. The face feature amounts also include secondaryinformation generated from pieces of individual information such as theratio of the interval between both eyes to the outline. These pieces ofinformation will be called feature amount elements. The face featureamounts serving as vector quantities formed from the feature amountelements are used to collate a face image in next step S404. Forexample, a structure as shown in FIG. 7 is prepared to record theextracted face feature amounts. FaceInfo is an area where the extractedface feature amounts are recorded. ImageID is an original imageincluding the face whose face feature amounts have been extracted.FaceID, ReferenceCount, Activation, NameInfo, and Property will bedescribed later.

In step S404, the image sensing unit 203 acquires a similarity bycomparing the face feature amounts acquired in step S403 with facefeature amounts registered in advance in the database unit 202 called aface dictionary (similarity determination unit).

The face dictionary will be described in detail. To perform personalrecognition, there is proposed a method of comparing newly obtained facefeature amounts with existing face feature amounts. A database whichrecords the existing face feature amounts is called a face dictionary.The recorded face feature amounts need to be discriminated from otherface feature amounts in the face dictionary. Therefore, at the time ofregistration in the face dictionary, the usage and identifier are addedto the face feature amounts acquired in step S403. In the embodiment,FaceID is the identifier for identifying the face feature amounts.ReferenceCount, Activation, NameInfo, and Property indicate the usage ofthe face feature amounts. These elements will be explained when actuallydescribing an example of use.

The face feature amounts to which the usage and identifier of thefeature amounts are added will be called an entry and will be simplyreferred to as an “entry”. FIG. 7 shows the entry. If there are aplurality of entries which are registered in the face dictionary and areto be compared, a plurality of similarities are calculated. A case inwhich an entry to be compared is not registered will be described later.In actual personal recognition, face feature amounts included in entriesare compared.

The similarity used to compare face feature amounts will be explained.The face feature amount comparison method is not particularly limited,and a known method is usable. For example, in Japanese Patent Laid-OpenNo. 2003-187229, the distance between feature amount vectors isevaluated as the similarity by using reliability based on a statisticalmethod. Japanese Patent Laid-Open No. 2006-227699 discloses a method ofevaluating, as the similarity, a calculated distance between featureamount vectors, though the feature amount vector calculation methoddiffers from that in the present invention. The method of evaluating thesimilarity between face feature amounts can be a known method, asdescribed above. In any case, the similarity is obtained based on thecomparison between face feature amounts.

Note that face feature amounts of the same types can be extracted fromall images so that they can be compared. However, even if the vectordimensions do not completely coincide with each other, these facefeature amounts may be evaluated by an operation of, for example,weighting obtained vector elements. A similarity exhibiting a smallestvalue is selected from the obtained similarities, and the processadvances to step S405. At this time, a feature amount counter used tocalculate the selected similarity is incremented by one. Morespecifically, the contents of ReferenceCount shown in FIG. 7 areupdated. The counter update method is not particularly limited as longas the record is finally left in a face feature amount used indetermination of personal recognition.

In step S405, the image sensing unit 203 evaluates the similarityacquired in step S404. If the similarity is higher than a set similarity(YES in step S405), the process advances to step S406. If the similarityis equal to or lower than the set similarity (NO in step S405), theprocess advances to step S407. If none of the face feature amounts to becompared is registered, the process advances to step S407.

The set similarity will be described. The set similarity is a thresholdfor evaluating a similarity obtained as a result of calculation. Thethreshold may be an internally permanent value. Alternatively, the usermay input the threshold by operating a UI (not shown). The threshold maychange in accordance with a learning states acquired from a useroperation. That is, the threshold can be set in an arbitrary form.

In step S406, the image sensing unit 203 additionally registers, in anexisting user group, face feature amounts determined to have asimilarity higher than the existing one. Details of the user group willbe explained in step S407.

In contrast, face feature amounts determined to have a similarity equalto or lower than the set one mean that the detected face imagerepresents a new person. In step S407, the image sensing unit 203registers these determined face feature amounts as a new user group.

The user group will be explained. Face images identified as face featureamounts having a high similarity are highly likely to represent the sameperson. Such images (objects) can be grouped and classified for eachobject, and displayed to the user. FIG. 8 exemplifies this display. Afolder window 801 displays persons in a tree-shaped structure. If theuser selects a person 802, a preview window 804 displays thumbnails 803of images. To implement this user friendliness, the user group isintroduced.

An example of a user group registration method will be explained. Morespecifically, NameInfo shown in FIG. 7 is used. If a person isdetermined as a new person, a new user group is created. Morespecifically, a new identifier identifiable by NameInfo is issued. Theidentifier will be explained. For example, assume that a person includedin an image is similar to none of n persons registered in the past. Inthis case, “n+1” meaning the (n+1)th person is described in NameInfo. Inthis manner, a new user group is created.

To “register in existing user group” in step S406, the assignedidentifier is described in the feature amount element NameInfo among thetarget face feature amounts. Note that the user group registrationmethod is not limited to this, and for example, the user may directlydesignate a character string. Alternatively, a table which associatesNameInfo with a character string designated by the user may be preparedseparately.

By registering the user group, images having the same NameInfo can begrouped and listed. In the embodiment, the system automatically issuesthe initial value of NameInfo. After that, the user can give a name, asneeded. The name is a name designated by the user for the person 802 inFIG. 8. The correspondence between the name designated by the user andthe user group may be managed in another table. NameInfo can also bedirectly rewritten.

In step S408, the image sensing unit 203 registers, in the facedictionary, the entry including the face feature amounts extracted instep S403. In this way, the entry (for each identifier) in the facedictionary is added. FIG. 9 schematically shows the face dictionarygenerated by registering a plurality of face feature amounts. The entrycan be created by additionally writing it in the above-described facedictionary.

In step S409, the image sensing unit 203 evaluates another person imagein image data. If there is another image to be processed (YES in stepS409), the process returns to step S401. If there is no image to beprocessed (NO in step S409), the process advances to step S410. In stepS410, the image sensing unit 203 determines whether all image data inthe folder have been processed. If all image data have not beenprocessed (NO in step S410), the process returns to step S400. If allimage data have been processed (YES in step S410), the process ends.Accordingly, personal recognition of a person is executed simultaneouslywhen a face dictionary is generated.

Next, a method of customizing the thus-generated face dictionary will beexplained. FIG. 10 is a flowchart showing face dictionary customizationprocedures.

In step S1000, for each user group, the image sensing unit 203 extractsImageID included in the user group. Table 3 exemplifies the extractionresults. Table 3 lists ImageIDs included for each user group identifiedby NameInfo.

TABLE 3 NameInfo ImageID 0 0 1 2 3 4 5 6 7 1 0 3 4 7 8 9 2 1 4 5 8 9 103 1 2 3 5 8 4 11

For example, a user group identified by NameInfo=0 includes ImageID=0,1, 2, 3, 4, 5, 6, 7. A user group identified by NameInfo=1 includesImaeID=0, 3, 4, 7, 8, 9.

In step S1001, the image sensing unit 203 counts the number of ImageIDsshared between different user groups. For example, ImageID=0, 3, 4, 7are shared between the user group identified by NameInfo=0 and the usergroup identified by NameInfo=1. In other words, NameInfo=0 andNameInfo=1 recognized as different persons simultaneously exist (arecaptured) in images of ImageID=0, 3, 4, 7. This reveals that the numberof images shared between NameInfo=0 and NameInfo=1 is four. The numberof shared images will be called “sharing level”. An example of theexpression is “the sharing level between NameInfo=0 and NameInfo=1 isfour”. Similarly, sharing levels between all different user groups arechecked. For each user group, sharing levels with other user groups arecounted. Table 4 exemplifies sharing levels counted between all usergroups.

TABLE 4 NameInfo 0 1 2 3 4 Total 0 4 3 4 0 11 1 4 3 2 0 5 2 3 3 3 0 6 34 2 3 0 5 4 0 0 0 0 0

In step S1002, the image sensing unit 203 calculates the sum of sharinglevels counted for each user group. The rightmost column of Table 4shows the sum of sharing levels.

In step S1003, the image sensing unit 203 evaluates the sum of sharinglevels calculated in step S1002. In this case, the sum of sharing levelsis compared with a predetermined threshold, and it is determined whetherthe user group shares a predetermined number (threshold) or more ofimages. If it is determined that the sum is equal to or smaller than thepredetermined threshold (NO in step S1003), the process advances to stepS1004. If it is determined that the sum is larger than the predeterminedthreshold (YES in step S1003), the process advances to step S1005. Thethreshold may be set in advance or set by the user.

In step S1004, the image sensing unit 203 regards all face featureamounts forming the user group as unnecessary face feature amounts. Atthis time, the image sensing unit 203 sets a flag (ignorance flag (useinhibition flag)) not to use these face feature amounts at the time ofsensing. More specifically, the flag is set in Activation of FIG. 7. Forexample, it is defined that the sum of sharing levels has reached apredetermined use frequency when Activation is “0”, and has not reachedit when Activation is “1”. The initial value of Activation is set to“0”. If the acquired sum of sharing levels has reached the predeterminedthreshold, Activation remains the initial value. In the use of the facedictionary, it can be inhibited to refer to the entry of a user groupwhich does not share a predetermined number or more of images.

In step S1005, the image sensing unit 203 determines whether the sum ofsharing levels has been evaluated for all user groups. If the sum ofsharing levels has not been evaluated for all user groups (NO in stepS1005), the process returns to step S1003. If the sum of sharing levelshas been evaluated for all user groups (YES in step S1005), the processends.

In the embodiment, the sharing level between user groups is employed asonly one evaluation scale to determine whether to use an entry whenusing the face dictionary. However, the number of evaluation scales isnot limited to one. For example, a case in which the number of imagedata is small at the initial stage of face dictionary generation will beexplained. At this time, if the face dictionary is customized using onlythe sharing level between user groups as the evaluation scale, referenceof all registered face feature amounts is always inhibited depending onthe predetermined threshold. To prevent this situation, the ratio of thetotal number of images belonging to a user group to the number of imagesshared between different user groups may be used as an evaluation scaleto determine whether to use an entry when using the face dictionary.That is, this ratio is used as the second evaluation scale, and if thenumber of shared images is equal to or larger than a predeterminedratio, the evaluation in step S1003 can be skipped.

By the above-described processing, the customization of the facedictionary ends. An entry determined not to be referred to will not beused, and the accuracy of the face dictionary can be increased.

As described above, according to the first embodiment, when using theface dictionary, whether to use an entry is determined in accordancewith the sharing level of images between user groups. More specifically,if the sharing level between user groups is equal to or lower than apredetermined criterion, an entry is not used in the use of the facedictionary. Hence, the use of an entry registered by incorrectdetermination or erroneous determination among entries registered in theface dictionary can be prevented. This can increase the recognitionaccuracy of the face dictionary.

Second Embodiment

The second embodiment will explain another form of face dictionarycustomization in the first embodiment. The hardware arrangement, thesoftware arrangement, and generation of the face dictionary are the sameas those in the first embodiment, and a description thereof will not berepeated.

An outline of processing in the second embodiment will be described. Aperson of high interest as an object is often in focus. It is thereforedetermined to register the face image of an in-focus object as a facefeature amount in the face dictionary. To the contrary, a person of lessinterest as an object is often out of focus. Further, an out-of-focusimage of even an object of high interest cannot represent the feature ofthe object correctly. Hence, it is determined not to register anout-of-focus image in the face dictionary.

A method of evaluating a focusing level will be described. The method ofevaluating a focusing level is not particularly limited. For example,there is a method of intentionally smoothing a target image andevaluating a change of a pixel before and after smoothing. If the targetimage is in focus, the change of the pixel upon smoothing is expected tobe large. If the target image is out of focus, the change upon smoothingis expected to be small. The focusing level is evaluated based on thedifference in expected change amount. This method will be described indetail with reference to FIG. 11.

FIG. 11 is a flowchart showing face dictionary customization proceduresaccording to the second embodiment.

In step S1100, an image sensing unit 203 acquires an original face imagefrom which face feature amounts have been extracted. At this time, theresolution of a face image to be acquired is arbitrary. The resolutionof the original image may be unchanged or changed. In any case, itsuffices to acquire an image in which the face region is trimmed. Forexample, coordinate positions are acquired from structured/describedimage sensing information. At this time, the resolution of the faceimage to be acquired can also be acquired.

In step S1101, the image sensing unit 203 applies a smoothing filter tothe trimmed image acquired in step S1100. FIG. 12 shows an example ofthe smoothing filter used to smooth an image in terms of the spatialfrequency of the image. The sharpness of the original image can bereduced by applying the smoothing filter to the trimmed image.

In step S1102, the image sensing unit 203 acquires the change amount ofa pixel upon smoothing from images before and after applying thesmoothing filter in step S1101. More specifically, the absolute value ofthe difference between signal values of the pixel that indicate the samecoordinates in images before and after smoothing is calculated, and thesum is calculated for all pixels. As a result, the change amount used tomeasure the degree of change in the entire region of the acquired faceimage is obtained.

In step S1103, the image sensing unit 203 evaluates the obtained changeamount. If the change amount is equal to or larger than a predeterminedthreshold (YES in step S1103), it is determined that the original imagehas a satisfactory sharpness (sharpness is equal to or higher than thethreshold). In this case, the process advances to step S1105 withoutchanging the face feature amounts. In contrast, if the change amount issmaller than the predetermined threshold (NO in step S1103), it isdetermined that the original image does not have a satisfactorysharpness, and the process advances to step S1104.

In step S1104, the image sensing unit 203 sets a flag (ignorance flag)to ignore the target face feature amounts. It suffices to set the flagin Activation, similar to the first embodiment.

In step S1105, the image sensing unit 203 determines whether all facefeature amounts have been evaluated. If there is an undetermined facefeature amount (NO in step S1105), the process returns to step S1100. Ifthere is no undetermined face feature amount (YES in step S1105), theprocess ends.

As described above, according to the second embodiment, an image whoseimproper face feature amounts have been registered in the facedictionary is specified in terms of the spatial frequency. If it isdetermined that the face feature amounts are improper, the use of themis restricted, thereby increasing the recognition accuracy of the facedictionary.

Third Embodiment

The third embodiment will exemplify the timing to customize the facedictionary. The hardware arrangement, the software arrangement, andgeneration of the face dictionary are the same as those in the firstembodiment, and a description thereof will not be repeated.

The timing to customize the face dictionary is not particularly limited.This timing will be explained by exemplifying a case in which the facedictionary is generated for the first time.

FIG. 13 is a flowchart showing personal recognition processing includingface dictionary customization. When generating a face dictionary for thefirst time, an image sensing unit 203 executes face dictionarycustomization in step S1311, as shown in FIG. 13, after step S410 of theflowchart of FIG. 4 in the first embodiment. As a result, face featureamounts which are erroneously registered in initial registration of theface dictionary can be excluded. Step S1311 is, for example, processingdescribed with reference to FIG. 10 or 11. This processing may beexecuted before step S400 as part of initialization when newly reading afolder unless the face dictionary is generated for the first time.Customization can also be executed if a predetermined condition issatisfied.

Next, another example of execution trigger generation will be explained.FIG. 14 is a flowchart showing a state in which customization isexecuted when a predetermined condition is satisfied.

In step S1400, the image sensing unit 203 determines whether the usageof the face dictionary satisfies a predetermined condition. Thepredetermined condition is not particularly limited and is, for example,whether the use time has exceeded a predetermined time. In theembodiment, it is determined whether the time (use time) elapsed aftercustomization was executed last time has exceeded a predetermined time.This information is not stored as a characteristic amount or facefeature amount, but can be stored as part of system control in asecondary storage apparatus 103. At the same time, a threshold todetermine whether to execute customization is also recorded. Whether thepredetermined condition is satisfied is determined by evaluating thesetwo amounts.

If it is determined that the predetermined condition is not satisfied(NO in step S1400), the process ends. If the predetermined condition issatisfied (YES in step S1400), the process advances to step S1401. StepS1401 is processing described with reference to FIG. 10 or 11.

An example of the predetermined condition is whether the number ofimages has exceeded a predetermined value. That is, customization of theface dictionary is executed when the number of images read aftercustomization was executed last time exceeds the predetermined value. Inthis case, the number of images read after executing customization ofthe face dictionary is recorded. Another example of the predeterminedcondition is whether the number of face feature amounts has exceeded apredetermined value. That is, customization of the face dictionary isexecuted when the number of face feature amounts registered in the facedictionary after customization was executed last time exceeds thepredetermined value. In this case, the number of face feature amountsregistered in the face dictionary is recorded.

Further, customization can be executed based on an instruction from theuser. In this case, software is instructed to execute customization viaa dedicated UI (not shown). The customization need not always beexecuted quickly in response to an instruction from the user. Forexample, it is also possible to temporarily store the instruction fromthe user in the RAM and process it at the end of the program.

As described above, according to the third embodiment, the facedictionary can be customized (for example, the ignorance flag can beset) at an appropriate timing, in addition to the effects described inthe above embodiments.

Fourth Embodiment

The fourth embodiment will exemplify a case in which an evaluation index(evaluation criterion) different from the sharing level in the firstembodiment is employed. The hardware arrangement, the softwarearrangement, and generation of the face dictionary are the same as thosein the first embodiment, and a description thereof will not be repeated.

FIG. 15 is a flowchart showing face dictionary customization proceduresaccording to the fourth embodiment.

In step S1500, an image sensing unit 203 acquires two different usefrequencies from use frequencies included in entries registered in theface dictionary. The use frequency will be explained here. Examples ofthe use frequencies of face feature amounts registered in the facedictionary are the number of times (number of times of use) by which theface feature amounts have actually been used for personal recognition,the time elapsed after they were used last time, and the time elapsedafter they were registered for the first time. Information of these usefrequencies is described in Property of FIG. 7. Table 5 exemplifies usefrequencies used in the embodiment.

TABLE 5 ReferenceCount Number of faces identified using this entry forpersonal recognition LastAccess Date & time when this entry was usedlast time AccessInterval Time elapsed after this entry was used lasttime Preference Sum of favorability evaluation values associated withimage

The use frequencies may be selected after acquiring the overall entry,or only the use frequencies may be acquired.

In step S1501, the image sensing unit 203 determines whether the firstone of the acquired use frequencies has reached a predetermined usefrequency. In the embodiment, AccessInterval is used as the first one ofthe acquired use frequencies. In this case, the evaluation index used instep S1501 is the time (non-use time) elapsed after the face featureamounts were used last time for personal recognition. The predetermineduse frequency can be designated in advance in the program. As anothermethod, a threshold can be set from a UI (not shown). In any case, it isonly necessary to prepare a threshold for evaluating acquiredAccessInterval. If the acquired AccessInterval is equal to or shorterthan a predetermined elapsed time (NO in step S1501), the processadvances to step S1504. If the acquired AccessInterval is longer thanthe predetermined elapsed time (YES in step S1501), the process advancesto step S1502.

In step S1502, the image sensing unit 203 determines whether the secondone of the acquired use frequencies has reached a predetermined usefrequency. In the embodiment, the second one of the acquired usefrequencies is ReferenceCount. In this case, the evaluation index usedin step S1502 is the number of times by which the face feature amounthas been used for personal recognition. The predetermined use frequencycan be designated in advance in the program. As another method, athreshold can be set from a UI (not shown). In any case, it suffices toprepare a threshold for evaluating acquired ReferenceCount.

If the acquired ReferenceCount has not reached the predetermined usefrequency (NO in step S1502), the process advances to step S1503. Instep S1503, the image sensing unit 203 sets a flag not to use the facefeature amounts at the time of sensing. More specifically, the flag isdescribed in Activation of FIG. 7. If the acquired ReferenceCount hasreached the predetermined use frequency (YES in step S1502), the processadvances to step S1504.

In step S1504, the image sensing unit 203 determines whether all entriesin the face dictionary have been evaluated. If there is an undeterminedentry (NO in step S1504), the process returns to step S1500. If allentries have been determined (YES in step S1504), the process ends. Inthis way, the face dictionary can be customized.

The embodiment employs AccessInterval and ReferenceCount as the usefrequencies of an entry to be used for customization of the facedictionary. However, the use frequencies used to evaluate face featureamounts included in an entry are not limited to this pair of two usefrequencies, and may be appropriately selected in accordance with, forexample, the user's taste.

A case in which FavoriteRate is used as the use frequency of an entry tobe used for customization of the face dictionary will be explained. Theevaluation index used in this case is “favorability” set by the userfrom a UI. First, the user selects a thumbnail image 1600 he wants on aUI as shown in FIG. 16 by using a mouse pointer 1601. Then, the userright-clicks to display a dialog capable of inputting the favorite rate.The user can select the number of ★s in the menu in accordance with hisfavorability. In general, it is set to increase the number of ★s as“favorability” is higher. In this manner, the user can add“favorability” to an image he likes. By FaceID, the designated“favorability” can be reflected in an entry associated with the image.

More specifically, Preference in Table 5 is acquired from Property ofFIG. 7. Preference will be explained. Preference is an evaluation indexwhich reflects “favorability” given to a plurality of images associatedwith the entry. For example, Preference can be the total number of ★s of“favorability” associated with the entry. As another case, the ratio ofthe number of ★s to the number of associated images may be calculated.Alternatively, Preference itself may be divided into five levels todesignate a level for every total number of ★s. It suffices to reflectthe “favorability” of an image in the entry.

At this time, the evaluation index used in step S1501 or S1502 isPreference of the entry. Even when this evaluation index is used, apredetermined use frequency can be designated in advance in the program.As another method, a threshold can be set from a UI (not shown). In anycase, it is only necessary to prepare a threshold for evaluatingacquired Preference.

As another use frequency used as an evaluation index in step S1501 orS1502, the time elapsed after the entry was registered for the firsttime can be used. Alternatively, the number of times by which an imageassociated with the entry has been printed can be recorded in the entry,like Preference. The number of times by which an image associated withthe entry has been enlarged and displayed in a preview window (FIG. 8 orthe like) may be recorded in the entry. Further, the number of accessesof the file of an image associated with the entry may be counted andrecorded in the entry.

Note that the embodiment adopts two use frequencies of face featureamounts used to customize the face dictionary. However, the usefrequencies used to evaluate face feature amounts are not limited totwo. For a finer condition determination, three or more use frequenciesare also available. Alternatively, only one use frequency is usable forsimplicity.

As described above, according to the fourth embodiment, the use of anentry having a low use frequency is excluded, thereby increasing therecognition accuracy of the face dictionary. Since the face dictionaryis customized in accordance with the use frequency of each entry of theface dictionary, a face dictionary considering the record of use by theuser can be built.

Fifth Embodiment

The fifth embodiment will exemplify a modification of generation andusage of the face dictionary in the first embodiment. The hardwarearrangement, the software arrangement, and generation of the facedictionary are the same as those in the first embodiment, and adescription thereof will not be repeated.

Face dictionary generation step S300 will be described in detail withreference to FIG. 17.

Note that steps S1700 to S1703 in FIG. 17 have the same processingcontents as those of steps S400 to S403 in FIG. 4, and a descriptionthereof will not be repeated.

In step S1704, an image sensing unit 203 acquires a similarity betweenface feature amounts. In the embodiment, the similarity is obtained bycomparing an acquired face feature amount with a “face feature amountderived by a predetermined procedure from a plurality of face featureamounts for identifying a person”. A set of face feature amounts foridentifying a person will be referred to as a “category”. The “facefeature amount derived by a predetermined procedure from the category”will be referred to as a “virtual face feature amount”. The categorywill be described later. The similarity is the same as that in the firstembodiment, and a description thereof will not be repeated. If there area plurality of categories to be compared that are registered in adatabase unit 202, a plurality of similarities are calculated.

In step S1705, the image sensing unit 203 evaluates the similarityobtained in step S1704. It is determined whether there is a categorywhich holds similar face feature amounts. If there is a category whichholds similar face feature amounts, that is, the similarity acquired instep S1704 is higher than a predetermined similarity (YES in stepS1705), the process advances to step S1706. If there is no categorywhich holds similar face feature amounts, that is, the similarityacquired in step S1704 is equal to or lower than the predeterminedsimilarity between all categories (NO in step S1705), the processadvances to step S1710. If none of the face feature amounts to becompared is registered, the process advances to step S1710. Thepredetermined similarity has been explained in the first embodiment, anda description thereof will not be repeated.

In step S1706, the image sensing unit 203 determines whether thecategory determined to have a similarity higher than the predeterminedsimilarity is associated with an existing user group. The user group hasbeen explained in the first embodiment, and a description thereof willnot be repeated. If the category is associated with an existing usergroup (YES in step S1706), the process advances to step S1707. If thecategory is not associated with an existing user group (NO in stepS1706), the process advances to step S1708.

In step S1707, the image sensing unit 203 registers the face featureamounts in the existing user group. The registration method has beenexplained in the first embodiment, and a description thereof will not berepeated.

In step S1708, the image sensing unit 203 determines whether thecategory has a blank. Here, the category will be described. The categoryis a set of face feature amounts for identifying a person, in otherwords, a unit for handling a finite number of face feature amounts in apredetermined similarity range. More specifically, when recognizing aperson for whom a specific face feature amount has been extracted, thecategory manages up to a finite number of (for example, a maximum offive) face feature amounts similar to the face feature amount in apredetermined range. The finite number may be set by giving apredetermined value, or selected and set by the user. Face featureamounts can be selected by associating FaceID with the category.

As described above, a plurality of face feature amounts are used becausethe face feature amounts of even the same person vary depending on howthe person is captured in the image. To raise the recognition tolerance,face feature amounts considering the range of variations are used. Toconsider the range of variations, face feature amounts calculated from aplurality of face feature amounts are used. The face feature amountcalculated from a plurality of face feature amounts serves as theabove-mentioned virtual face feature amount. The virtual face featureamount can be used to reduce the risk of erroneous determination arisingfrom personal recognition using one arbitrary.

An example of the structure of the category will be explained withreference to FIG. 18.

CategoryID is an identifier for identifying each category.ReferenceCount, Activation, NameInfo, and Property are the same as thoseof FIG. 7 in the first embodiment, and a description thereof will not berepeated. IdealFeature will be explained. The virtual face featureamount is recorded in IdealFeature. The virtual face feature amountcalculation method will be described. An example of the virtual facefeature amount for calculating a similarity can be the average of facefeature amounts associated with the category. Note that the virtual facefeature amount is not limited to the average of face feature amounts,and only face feature amounts having maximum values may be collected tocalculate the average. The average or median may be calculated based onthe variance of feature amount elements.

FIG. 18 exemplifies a structure in which a maximum of five FaceIDs areassociated with the category. As described above, the category is a setof a finite number of face feature amounts. Hence, associable FaceID canbe registered unless the number of FaceIDs has reached a set maximumvalue. If the number of FaceIDs has reached a set maximum value, noassociable FaceID can be registered. In step S1708, to evaluate a marginof registration, the image sensing unit 203 determines whether thecategory has a blank. If the category has a blank (YES in step S1708),the process advances to step S1709. If the category has no blank (NO instep S1708), the process advances to step S1712.

Although the embodiment uses FaceID to associate a category with a facefeature amount, the present invention is not limited to this. Forexample, a category and entry may be associated in another database, andthis database may be referred to.

In step S1709, the image sensing unit 203 associates the target FaceIDwith the existing category to update the category.

A face feature amount determined to have a similarity equal to or lowerthan the predetermined similarity cannot be registered in any category.In step S1710, therefore, the image sensing unit 203 creates a newcategory for the face feature amount, and associates FaceID with it.More specifically, a new code is assigned to the category ID.

In step S1711, the image sensing unit 203 newly creates a user group.Creation and management of the user group have been explained in thefirst embodiment, and a description thereof will not be repeated.

Subsequent steps S1712 to S1714 correspond to steps S408 to S410 in thefirst embodiment, their processing contents are the same as thosedescribed above, and a description thereof will not be repeated.

Customization of the thus-generated face dictionary will be explainedwith reference to FIG. 19. Only a difference from the first embodimentwill be described. FIG. 19 is a flowchart showing face dictionarycustomization procedures.

The processing contents of steps S1900 to S1903 are the same as those ofsteps S1000 to S1003 in FIG. 10, and a description thereof will not berepeated.

In step S1904, the image sensing unit 203 sets a flag (ignorance flag)so that all entries forming the target category are not used at the timeof sensing. More specifically, the flag is set in Activation of FIG. 7.For example, it is defined that the sum of sharing levels has reached apredetermined use frequency when Activation is “0”, and has not reachedit when Activation is “1”. The initial value of Activation is set to“0”. If the acquired ReferenceCount has reached the predetermined usefrequency, Activation remains the initial value.

Note that customization of a category and customization of face featureamounts may coexist. In this case, the following rule can be set inadvance. For example, when already evaluated face feature amounts areclassified into a category later, the subsequent reference count andusage of them in the category are preferentially considered.Alternatively, the use frequency of the face feature amounts may becleared after classification into the category. In any case, confusioncan be avoided by making a rule for the usage.

In step S1905, the image sensing unit 203 determines whether allcategories in the face dictionary have been evaluated. If all categorieshave not been evaluated (NO in step S1905), the process returns to stepS1900. If all categories have been evaluated (YES in step S1905), theprocess ends. In this fashion, the face dictionary can be categorized.Note that the evaluation scale to be used is not limited to the usefrequency. Also, the customization method can use a plurality of usefrequencies, as described in the third embodiment. Another example ofthe use frequency has been described in the first embodiment, and adescription thereof will not be repeated.

The difference between the fifth embodiment and the first embodiment iswhich of the face feature amount or virtual face feature amount is used.The fifth embodiment can be implemented in combination with the secondor third embodiment.

As described above, according to the fifth embodiment, face featureamounts in a predetermined similarity range are managed as one categorybelonging to each person. A face dictionary considering states of aperson such as various expressions can be built.

Sixth Embodiment

The sixth embodiment will describe a method of designating a categorynot to be used in dictionary customization. The hardware arrangement andsoftware arrangement in the sixth embodiment are the same as those inthe first embodiment, and a description thereof will not be repeated.

FIG. 20 is a flowchart showing face dictionary customization proceduresaccording to the sixth embodiment.

In step S2000, an image sensing unit 203 acquires a system status. Thesystem status is, for example, the remaining capacity of a secondarystorage apparatus 103, or the total number of categories registered in adatabase unit 202. The image sensing unit 203 acquires such a value, anddetermines whether the acquired value is larger than a predeterminedvalue. If the image sensing unit 203 determines that the acquired valueis equal to or smaller than the predetermined value (YES in step S2000),the process ends. If the image sensing unit 203 determines that theacquired value is larger than the predetermined value (NO in stepS2000), the process advances to step S2001.

In step S2001, the image sensing unit 203 acquires and evaluates a usefrequency. As the use frequency to be acquired, NameInfo in FIG. 7 isused.

In step S2002, the image sensing unit 203 evaluates the use frequencyacquired in step S2001. In the embodiment, if NameInfo remains theinitial value, the image sensing unit 203 determines that nopredetermined use frequency is satisfied (NO in step S2002), and theprocess advances to step S2003. Note that it can be determined that theuser has little interest in the target person when the predetermined usefrequency is not satisfied. If the image sensing unit 203 determinesthat the predetermined use frequency is satisfied (YES in step S2002),the process advances to step S2004. Note that it can be determined thatthe user is interested in the target person when the predetermined usefrequency is satisfied.

In step S2003, the image sensing unit 203 erases the target category.

Step S2004 has the same processing contents as those of step S1905 inthe fourth embodiment, and a description thereof will not be repeated.

As described above, according to the sixth embodiment, the accuracy ofthe face dictionary can be increased by erasing an entry which does notsatisfy a predetermined use frequency. The size of the face dictionarycan be reduced by erasing, of the entries of the face dictionary, anentry which does not satisfy the predetermined use frequency. Bydecreasing the number of entries, the face dictionary reference countaccompanying personal recognition can be reduced to increase theprocessing speed.

Other Embodiments

The above-described embodiments are examples for obtaining the effectsof the present invention. If the same effects as those of the presentinvention are obtained by using another similar method or differentparameters, this also falls within the scope of the invention.

Although the use of a predetermined entry is inhibited by setting theflag in the first to fifth embodiments, the predetermined entry may bedeleted from the face dictionary. This can reduce the size of the facedictionary. By decreasing the number of entries, the face dictionaryreference count accompanying personal recognition can be reduced toincrease the processing speed.

The above-described embodiments have explained, as a layout output, anoutput in which a plurality of images are arranged on one page. However,the present invention is also applicable to output of a plurality ofpages.

The above-described embodiments have explained a case in whichgeneration and collation of the face dictionary are performedsimultaneously, but generation and collation of the face dictionary maybe performed separately. For example, either the face dictionarygeneration mode or collation mode may be used separately in folderscanning. In this case, when a folder is scanned using the facedictionary generation mode, the processing can skip steps S405 to S407in FIG. 4 and be executed. To the contrary, when a folder is scannedusing the collation mode, the processing can skip step S408 in FIG. 4and be executed. In this case, two scans in the face dictionarygeneration mode and collation mode are necessary when performingpersonal recognition for an image included in the folder for which theface dictionary has been generated.

Although the above-described embodiments have exemplified a person'sface as an object, the object is not limited to a person's face. A pet'sface can be set as an object by performing recognition processing for apet such as a dog or cat to recognize it. Since even a building, smallitem, or the like can be recognized by recognizing a shape by edgedetection or the like, it can also be set as an object. In these cases,image processing can be performed by the same method as those in theabove-described embodiments by extracting the feature amounts of anobject and registering them in the dictionary.

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (for example, non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment(s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blue-ray Disc(BD)′M), a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2012-154010, filed Jul. 9, 2012, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: atleast a processor and at least a memory coupled to the at least theprocessor and having stored thereon instructions which when executed bythe at least the processor cooperate to act as: an extraction unitconfigured to extract a feature information from an object in inputimage data; a registration unit configured to classify a plurality offeature information into a plurality of groups based upon a similarity,and register the plurality of feature information into a dictionary,wherein the dictionary is used to determine a similarity between thefeature information registered in the dictionary and the featureinformation extracted by the extraction unit; and a deletion unitconfigured to delete, among the plurality of groups, a featureinformation of a group, which does not reach a predetermined use level,from the dictionary, wherein information related to a time periodelapsed from when the feature information had been registered in thedictionary at first is used as the use level, and wherein, in a casewhere the number of feature information registered in the dictionaryexceeds a predetermined number, the deletion unit deletes, from thedictionary, the feature information of the group which does not reachthe predetermined use level.
 2. The apparatus according to claim 1,further comprising a determination unit configured to determine thesimilarity by referring to the dictionary.
 3. The apparatus according toclaim 1, wherein the registration unit registers each of the pluralityof groups and image data including a region corresponding to an objectof feature information in the group in association with each other. 4.The apparatus according to claim 1, further comprising: an input unitconfigured to input image data; and a detection unit configured todetect an object based upon image data inputted by the input unit.
 5. Animage processing apparatus comprising: at least a processor and at leasta memory coupled to the at least the processor and having stored thereoninstructions, when executed by the at least the processor, andcooperating to act as: an extraction unit configured to extract afeature information from an object in input image data; a registrationunit configured to classify a plurality of feature information into aplurality of groups based upon a similarity, and register the pluralityof feature information into a dictionary, wherein the dictionary is usedto determine a similarity between the feature information registered inthe dictionary and the feature information extracted by the extractionunit; and a deletion unit configured to delete, among the plurality ofgroups, a feature information of a group, which does not reach apredetermined use level, from the dictionary, wherein informationrelated to a time period elapsed from when the feature information hadbeen registered in the dictionary at first is used as the use level, andwherein, in a case where a total number of categories, each of which isas a set of feature information for identifying a user, exceed apredetermined number, the deletion units deletes, from the dictionary,the feature information of a group which does not reach thepredetermined use level.
 6. An image processing apparatus comprising: atleast a processor and at least a memory coupled to the at least theprocessor and having stored thereon instructions which when executed bythe at least the processor cooperate to act as: an extraction unitconfigured to extract a feature information from an object in inputimage data; a registration unit configured to classify a plurality offeature information into a plurality of groups based upon a similarity,and register the plurality of feature information into a dictionary,wherein the dictionary is used to determine a similarity between thefeature information registered in the dictionary and the featureinformation extracted by the extraction unit; and a deletion unitconfigured to delete, among the plurality of groups, a featureinformation of a group, which does not reach a predetermined use level,from the dictionary, wherein a name of an identifier of the group whichcan be rewritten by a user is used as the use level, and wherein thedeletion unit deletes, from the dictionary, a group whose name of theidentifier is an initial value, as the group not reaching thepredetermined use level.
 7. The apparatus according to claim 6, furthercomprising a determination unit configured to determine the similarityby referring to the dictionary.
 8. The apparatus according to claim 6,wherein the registration unit registers each of the plurality of groupsand image data including a region corresponding to an object of featureinformation in the group in association with each other.
 9. Theapparatus according to claim 6, wherein, in a case where the number offeature information registered in the dictionary exceeds a predeterminednumber, the deletion unit deletes, from the dictionary, the featureinformation of a group which does not reach the predetermined use level.10. The apparatus according to claim 6, further comprising: an inputunit configured to input image data; and a detection unit configured todetect an object based upon image data inputted by the input unit. 11.An image processing method comprising: extracting a feature informationfrom an object in input image data; classifying a plurality of featureinformation into a plurality of groups based upon a similarity, andregistering the plurality of feature information into a dictionary,wherein the dictionary is used to determine a similarity between thefeature information registered in the dictionary and the featureinformation extracted in the extracting; and deleting, among theplurality of groups, a feature information of a group, which does notreach a predetermined use level, from the dictionary, whereininformation related to a time period elapsed from when the featureinformation had been registered in the dictionary at first is used asthe use level, and wherein, in a case where the number of featureinformation registered in the dictionary exceeds a predetermined number,the feature information of a group which does not reach thepredetermined use level is deleted from the dictionary.
 12. The methodaccording to claim 11, further comprising determining the similarity byreferring to the dictionary.
 13. The method according to claim 11,wherein in the registering, each of the plurality of groups and imagedata including a region corresponding to an object of featureinformation in the group are registered in association with each other.14. The method according to claim 11, further comprising: inputtingimage data; and detecting an object based upon the input image data. 15.An image processing method comprising: extracting a feature informationfrom an object in input image data; classifying a plurality of featureinformation into a plurality of groups based upon a similarity, andregistering the plurality of feature information into a dictionary,wherein the dictionary is used to determine a similarity between thefeature information registered in the dictionary and the featureinformation extracted in the extracting; and deleting, among theplurality of groups, a feature information of a group, which does notreach a predetermined use level, from the dictionary, whereininformation related to a time period elapsed from when the featureinformation had been registered in the dictionary at first is used asthe use level, and wherein, in a case where a total number ofcategories, each of which is as a set of feature information foridentifying a user, exceed a predetermined number, the featureinformation of a group which does not reach the predetermined use levelis deleted from the dictionary.
 16. An image processing methodcomprising: extracting a feature information from an object in inputimage data; classifying a plurality of feature information into aplurality of groups based upon a similarity, and registering theplurality of feature information into a dictionary, wherein thedictionary is used to determine a similarity between the featureinformation registered in the dictionary and the feature informationextracted in the extracting; and deleting, among the plurality ofgroups, a feature information of a group, which does not reach apredetermined use level, from the dictionary, wherein a name of anidentifier of the group which can be rewritten by a user is used as theuse level, and wherein in the deleting, a group whose name of theidentifier is an initial value is deleted from the dictionary, as thegroup not reaching the predetermined use level.
 17. The method accordingto claim 16, further comprising determining the similarity by referringto the dictionary.
 18. The method according to claim 16, wherein in theregistering, each of the plurality of groups and image data including aregion corresponding to an object of feature information in the groupare registered in association with each other.
 19. The method accordingto claim 16, wherein, in a case where the number of feature informationregistered in the dictionary exceeds a predetermined number, the featureinformation of a group which does not reach the predetermined use levelis deleted from the dictionary.
 20. The method according to claim 16,wherein, in a case where a total number of categories, each of which isas a set of feature information for identifying a user, exceed apredetermined number, the feature information of a group which does notreach the predetermined use level is deleted from the dictionary. 21.The method according to claim 16, further comprising: inputting imagedata; and detecting an object based upon the input image data.